MONITORING SYSTEM 



BACKGROUND OF THE INVENTION 

The present invention relates to an image processing 
technique for synthesizing images, taken by multiple cameras, 
through various types of processing including deformation and 
integration. More particularly, the present invention re- 
lates to a technique effectively applicable to a monitoring 
system as an aid for vehicle driving operations, for example. 

Recently, car-mounted display devices have been more and 
more popularized and the prices of video capturing devices, 
including video cameras, have been reduced drastically. Re- 
flecting such tendencies, a system usable as a safety check 
aid for a vehicle driver by allowing him or her to monitor 
the vehicle's surroundings using video cameras is now avail- 
able at a reasonable price and is being popularized now. 

A system as disclosed in Japanese Patent Application No. 
10-217261 is an exemplary vehicle surrounding monitoring sys- 
tem of that type. Specifically, the system combines together 
images that have been taken by multiple cameras mounted on a 
vehicle to produce a synthesized image, which looks as if the 
vehicle were seen from a virtual viewpoint located over the 
vehicle, and then presents the virtual image to the driver - 

FIG. 19 illustrates an exemplary configuration for a sys- 
tem of that type. As shown in FIG. 19, the system includes: 
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camera section 40 made up of multiple cameras 401; image proc- 
essing section 50 coupled to the camera section 40 via trans- 
mission lines 45; and display device 60. The image processing 
section 50 includes: buffer memories 501 associated with the 
5 respective cameras 401; mapping table 502 describing a rela- 
tionship between a synthesized image that should look as such 
to a virtual viewpoint and the images actually taken by the 
cameras 401; and image synthesizer 503. Each of the cameras 
401 included in the camera section 40 outputs an image signal 

10 on a field-by-field or frame-by-frame basis. The image signal 
output is transmitted through associated one of the transmis- 
sion lines 45 and then stored on associated one of the buffer 
memories 501. By reference to the data stored on the mapping 
table 502, the image synthesizer 503 combines together the im- 

15 age signals , read out from the buffer memories 501, to produce 
a synthesized image and present it on the display device 60. 
By looking at the synthesized image produced by the system 
shown in FIG. 19, the user (i.e., the driver of a vehicle) can 
know an exact positional relationship between his or her own 

20 vehicle and the vehicle's surroundings without taking the ac- 
tual positions of the cameras into consideration. Accord- 
ingly, this system allows the driver to park his or her 
vehicle much more safely, for example. 

The system of this type, however, has the following draw- 

25 backs . 
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As shown in FIG. 19, each of the transmission lines 45 is 
associated with one of the cameras 401. That is to say, the 
same number of transmission lines 45 as that of the cameras 
401 should connect the cameras 401 to the image processing 
5 section 50. Normally, the cameras 401 are mounted at various 
positions of a vehicle to produce a synthesized image repre- 
senting the vehicle's surroundings. Accordingly, the vehicle 
should be wired with the transmission lines 45 here and there. 
m As a result, it takes too much time and trouble to install 

m 10 this system on the vehicle. Also, such a system requires a 
60 great deal of maintenance to avoid failures, for example. 

i . 5 

yj Stated otherwise, to make the system more easily in- 
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stallable and maintainable, the number of transmission lines 
should be reduced by gettirfg one transmission line shared by 

15 multiple cameras. However, it usually takes a huge channel 
capacity to transfer the image data of each camera entirely. 
For that reason, it would be hard for a reduced number of 
transmission lines to cope with such a tall demand. That is 
to say, to reduce the number of transmission lines, there is 

20 no other choice than cutting down the amount of image data to 
be transferred. 

Also, the image processing section 50 needs to store that 
huge amount of image data that has been transferred from the 
cameras 401 on a field-by-field or frame-by-frame basis. 

25 Thus, each of the buffer memories 501 should have a great 
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storage capacity. 

Furthermore, although a synthesized image is produced 
from a plurality of images taken by multiple cameras, not all 
of each camera image is used for the image synthesis but each 
5 image contains some unnecessary parts* Moreover, even in an 
image area required for the image synthesis, some part of the 
area should show a resolution different from that of another 
during the image synthesis process. Accordingly, it is not 
always necessary to transmit the entire image data of each 
10 camera image as it is to the image processing section. 

SUMMARY OF THE INVENTION 

An object of this invention is providing a monitoring 
system that includes multiple cameras and an image processing 

15 section for producing a synthesized image from the images ta- 
ken by the cameras and that can drastically reduce the amount 
of image data to be transmitted without sacrificing the qual- 
ity of the synthesized image. 

Specifically, a monitoring system according to the pre- 

20 sent invention includes camera section, transmission path and 
image processing section. The camera section includes multi- 
ple cameras and image data cutdown means . The image data 
cutdown means reduces the amount of original image data rep- 
resenting camera images captured by the cameras. The camera 

25 section outputs the reduced image data. The transmission path 
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transmits the reduced image data that has been output from 
the camera section. The image processing section receives the 
reduced image data through the transmission path and produces 
a synthesized image from the reduced image data. The camera 
section or the image processing section includes cutdown mode 
selecting means for selecting a cutdown mode, in which the 
original image data for use in image synthesis has its amount 
cut down, for the image data cutdown means in accordance with 
a correspondence between the synthesized and camera images. 



10 According to the present invention, the image data cut- 

wy 



down means, provided for the camera section, reduces the 
|7§ amount of original image data in compliance with the cutdown 

s mode selected by the cutdown mode selecting means. Then, the 

o 

SJ camera section outputs the reduced image data onto the trans- 

O 

ly 15 mission path. Also, the cutdown mode selecting means selects 
the cutdown mode, in which the original image data for use in 
image synthesis has its amount cut down, in accordance with 
the correspondence between the synthesized and camera images. 
Accordingly, the image data, which has had its amount cut down 

20 in the cutdown mode selected in accordance with the correspon- 
dence between the synthesized and camera images, is transmit- 
ted through the transmission path. In this manner, the amount 
of camera image data to be transmitted can be reduced drasti- 
cally without sacrificing the quality of the resultant synthe- 

25 sized image. As a result, the transmission path used may have 
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a reduced channel capacity. So the transmission path is im- 
plementable as a smaller number of less expensive transmission 
lines or even as a wireless path. Thus, the transmission path 
is much easier to mount on a vehicle or requires maintenance 
5 much less often. In addition, a buffer memory needed for the 
image processing section may have a much reduced storage ca- 
pacity. 

In one embodiment of the present invention, the image 
processing section may selectively produce any of multiple 
10 types of synthesized images and change the types of the syn- 
thesized images to be produced. The cutdown mode selecting 
means may change the cutdown modes according to the type of 
the synthesized image to be produced by the image processing 
section. 

15 In another embodiment of the present invention, the cut- 

down mode selecting means may include a resolution specifier 
for specifying resolutions that should be necessary for re- 
spective areas of each said camera image for use in the image 
synthesis to produce the synthesized image. In that case, the 

20 image data cutdown means preferably compresses the original 
image data, representing the camera images for use in the im- 
age synthesis, according to the resolutions specified by the 
resolution specifier. In this particular embodiment, the im- 
age data cutdown means preferably compresses the original im- 

25 age data by discrete cosine transform. 
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In such an embodiment, the original image data has its 
amount reduced greatly by being compressed according to the 
resolutions that should be necessary for image synthesis, and 
then image data in the reduced amount is transmitted* Ac- 
5 cordingly, the transmission path may have a much smaller 
channel capacity. In addition, it is also possible to sup- 
press the aliasing distortion, thus improving the quality of 
the resultant synthesized image. 
m In an alternative embodiment, the cutdown mode selecting 

m 10 means may include an area specifier for specifying areas that 
m should be necessary to produce the synthesized image for each 

yj said camera image for use in the image synthesis. In that 

lira 

%p £ 

e case, the image data cutdown means preferably removes an unne- 

tsar 

•Nf cessary part from the original image data that represents each 

W 15 said camera image for use in the image synthesis. The unne- 
^ cessary part corresponds to the remaining area of the camera 

image other than the areas specified by the area specifier. 

In such an embodiment, the original image data has its 
amount reduced drastically before the transmission because 
20 the unnecessary part of the original image data, correspond- 
ing to the excessive area thereof other than the areas neces- 
sary for image synthesis, is removed. Accordingly, the 
transmission path may have a much smaller channel capacity. 

In yet another embodiment, the original image data may be 
25 read out from each said camera in an externally controllable 
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order. In that case, the camera section or the image process- 
ing section preferably includes a readout controller for con- 
trolling the order, in which the original image data 
representing each said camera image for use in the image syn- 
5 thesis is read out, in compliance with the cutdown mode se- 
lected by the cutdown mode selecting means. 

Then, the camera images for use in image synthesis will 
be read out in the order that has been controlled in compli- 
_ ance with the cutdown mode selected. Accordingly, it is pos- 

5? 10 sible to prevent too much data from being transmitted through 
:? the transmission path within a limited period of time. 

Rather, the image data can be transmitted dispersively . 

In yet another embodiment, the cameras are preferably 
mounted on a vehicle to capture images of the vehicle's sur- 
15 roundings . 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram illustrating a configuration 
for a monitoring system according to a first embodiment of 
20 the present invention. 

FIG. 2 illustrates an exemplary arrangement of cameras 
on a vehicle. 

FIG. 3 illustrates exemplary positions of virtual view- 
points for synthesized images. 
25 FIGS. 4A and 4B illustrate an exemplary correspondence 
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between respective camera images and a resultant synthesized 
image, 

FIGS. 5A and 5B schematically illustrate a correspon- 
dence between a synthesized image and a camera image. 
5 FIGS. 6A and 6B illustrate how the areas of a camera im- 

age may be classified according to resolutions. 

FIGS. 7A, 7B and 7C illustrate an exemplary method of 
deriving resolution data. 

FIG. 8 is a block diagram illustrating an exemplary in- 

I jj 

g 10 ternal configuration for a compressor according to the first 

yy 

embodiment . 

21™ 
\ .. H 

r% FIG. 9 illustrates another exemplary arrangement of cam- 

lr§ 

J" eras on a vehicle. 

FIG. 10 is block diagram illustrating, as a comparative 

O 

y 15 example, a monitoring system with an image processing section 
including LPFs . 

FIGS. 11A and 11B illustrate another exemplary method of 
deriving resolution data. 

FIG. 12 is a block diagram illustrating a configuration 
20 for a monitoring system according to a second embodiment of 
the present invention. 

FIG. 13 is a block diagram illustrating an exemplary in- 
ternal configuration for camera and compressor according to 
the second embodiment. 
25 FIG. 14 illustrates an exemplary order in which image 
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signals are read out from respective cameras. 

FIG. 15 is a block diagram illustrating a configuration 
for a monitoring system according to a third embodiment of 
the present invention. 

FIGS. 16A and 16B illustrate an area necessary for image 
synthesis . 

FIGS. 17A and 17B illustrate two methods of specifying 
an area necessary for image synthesis. 

FIG. 18 is a block diagram illustrating an exemplary in- 
ternal configuration for a selector according to the third 
embodiment . 

FIG. 19 is a block diagram illustrating a configuration 
for a known monitoring system. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Hereinafter, preferred embodiments of the present inven- 
tion will be described with reference to the accompanying 
drawings . 

EMBODIMENT 1 

FIG. 1 is a block diagram illustrating a configuration 
for a monitoring system according to a first embodiment of 
the present invention. The monitoring system shown in FIG. 1 
is supposed to be mounted on a vehicle and used as an aid for 
vehicle driving operations (e.g., parking). Specifically, 
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this system combines together images that have been taken by 
multiple cameras mounted on a vehicle to produce a synthe- 
sized image, which looks as if the vehicle were captured by a 
camera located over the vehicle, for example. Then, the sys- 
5 tern presents the virtual image to the driver. By looking at 
the synthesized image presented, the driver can know an exact 
positional relationship between his or her own vehicle and its 
surroundings, and can park his or her vehicle much more 
safely, for example. 

□ 

*0 10 As shown in FIG. 1, the monitoring system includes cam- 

jjf era section 10, transmission line 15, image processing see- 

dy 

Yi tion 20 and display device 30. The camera section 10 includes 

LP 

a number X of cameras 110 (i.e., cameras No. 1 through No. 

\j X), each of which is integrated together with a compressor 

□ 

Ixi 15 120 and a transmission adapter (Adp) 130. The compressor 120 
*H= compresses an image taken by its associated camera 110. The 

image data, representing the respective images taken by these 
cameras 110, is output from the camera section 10, transmit- 
ted through the transmission line 15 and then input to the 
20 image processing section 20. That is to say, the transmis- 
sion line 15 serves as a transmission path that connects the 
camera and image processing sections 10 and 20 together. On 
receiving the image data, the image processing section 20 
performs various types of processing (including deformation 
25 and integration) on the image data, thereby producing a syn- 
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thesized image and presenting it on the display device 30. 

FIG. 2 illustrates an exemplary arrangement of cameras 
in accordance with this embodiment. In the example illus- 
trated in FIG. 2, six cameras Nos . 1 through 6 are mounted at 
5 respective positions on a vehicle. FIG. 3 illustrates exem- 
plary positions of virtual viewpoints for synthesized images. 
The image processing section 20 produces a synthesized image, 
which looks as if the vehicle were seen from the virtual view- 
point No. 1 shown in FIG. 3, for example, from the images ta- 

o 

Jj=f 10 ken by the cameras Nos . 1 through 6 . 

* a H 
vr-=r 

r£ As shown in FIG. 1, the image processing section 20 in- 

^ eludes image synthesizer 210, mapping table 220, buffer memory 

230, expander 240, transmission adapter (Adp) 250 and resolu- 
Cj tion specifier 260. The mapping table 220 stores data de- 

yg 15 scribing a correspondence between a synthesized image and 
M> respective camera images. The image synthesizer 210 produces 

a synthesized image by reference to the mapping table 220. 
The compressed image data is input through the transmission 
adapter 250 and then decompressed by the expander 240. Then, 
20 the decompressed image data is temporarily retained on the 
buffer memory 230. In accordance with the correspondence be- 
tween the synthesized and camera images as described on the 
mapping table 220, the resolution specifier 260 specifies 
resolutions, which should be necessary to produce the synthe- 
25 sized image, for respective areas of each camera image. The 
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resolution specifier 260 is exemplary cutdown mode selecting 
means as defined in the appended claims. 

FIGS. 4A and 4B illustrate an exemplary correspondence 
between respective camera images and a resultant synthesized 
5 image. In the illustrated example, the synthesized image shown 
in FIG. 4A, which looks as if the vehicle were seen from the 
virtual viewpoint No. 1 shown in FIG. 3, is produced from the 
images taken by the cameras Nos . 1 through 6 arranged as shown 
in FIG. 2. As shown in FIG. 4A, the images taken by the cam- 

10 eras Nos. 1 through 6 will respectively occupy the areas CAl 
through CA6 of the synthesized image. In the following de- 
scription, the areas CAl through CA6 will be referred to as 
"camera image areas" CAl through CA6 . In FIG. 4B on the other 
hand, the camera image areas corresponding to the areas CAl 

15 through CA6 of the synthesized image are identified by CAl' 
through CA6', respectively. 

The mapping table 220 stores data representing the corre- 
spondence between the synthesized and respective camera images 
such as that shown in FIG. 4A. That is to say, the data 

20 stored on the mapping table 220 describes camera image data 
associated with every set of coordinates included in the syn- 
thesized image that should look as such to the virtual view- 
point. By reference to the data stored on the mapping table 
220, the image synthesizer 210 produces the synthesized image 

25 that looks as if the vehicle were seen from the virtual view- 
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For example, a point PI on the synthesized image is lo- 
cated in an area OA in which the image areas CAl and CA6 of 
the cameras Nos. 1 and 6 overlap with each other. Accordingly, 
5 pixel data is obtained for this point PI using pixel data as- 
sociated with the point Pi in the area CAl' of the camera No. 
1 image and pixel data associated with the point PI in the 
area CA6 ' of the camera No. 6 image. On the other hand, a 
point P2 on the synthesized image is located in the image area 



y3 10 CAl of the camera No. 1. Accordingly, pixel data is obtained 



Also, in accordance with the correspondence between the 
synthesized and respective camera images as described on the 
15 mapping table 220, the resolution specifier 260 specifies re- 
solutions that should be necessary for image synthesis for the 
respective areas of each camera image. The data, representing 
the resolutions specified for each camera image, is transmit- 
ted through the transmission adapter 250 and transmission line 



20 15 to a compressor 120 associated with the camera image. in 
accordance with the resolution data transmitted, the compres- 
sor 120 compresses the image data of the associated camera im- 
age. The compressed image data is transmitted through the 
transmission adapter 130 and transmission line 15 to the image 

25 processing section 20. In the image processing section 20, 




for the point P2 using pixel data associated with the point P2 



in the area CAl' of the camera No. 1 image. 
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the compressed image data is decompressed by the expander 240 
and then the decompressed image data is stored on the buffer 
memory 230. 

Also, the mapping table 220 stores multiple sets of map- 
5 ping table data that are associated with a plurality of virtu- 
al viewpoints as shown in FIG- 3. In response to a selection 
signal externally input, the mapping table 220 can select one 
of those sets of mapping table data for use in the image syn- 
thesis. In this manner, the image processing section 20 can 
10 selectively produce one of multiple types of synthesized im- 
ages and can change the types of the synthesized images to be 
produced. The selection signal may be applied according to 
the gear position or the speed of a running vehicle, for exam- 
pie. 

y] 15 When the selection signal is input, the previous set of 

G 

M> mapping table data that has been output by the mapping table 

220 is replaced with another. As a result, a different type 
of synthesized image is produced. On receiving the newly se- 
lected set of mapping table data, the resolution specifier 260 

20 specifies again the resolutions that should be necessary for 
image synthesis for respective areas of each camera image. 
The resolution data newly obtained is transmitted through the 
transmission line 15 to the compressor 120 associated with 
each camera image. And in accordance with the resolution data 

25 transmitted, the compressor 120 changes the modes of compres- 
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sion processing to be carried out. 

Compression processing 

In the illustrated embodiment, each of the compressors 
120 is supposed to compress the associated camera image data 
by discrete cosine transform (DCT) . 

For example, first, the compressor 120 divides its asso- 
ciated camera image made up of 480 X 720 pixels into 5400 mac- 
roblocks, each consisting of 8X8 pixels* As a result of this 
division, camera image data with coordinates (i, j) (where 1^ 
i^ 480, l^j^ 720) may be represented as SI (K, L, i', j'), 
where K and L respectively are horizontal and vertical block 
address numbers, i.e., 1^K^60, 1^L^90, l^i'^8 and 1 ^ j ' ^ 
8. 

Then, each pixel data SI (K, L, i', j') is transformed 
into a signal gl (K, L, m, n) with respective frequency compo- 
nents by the DCT transform given by 

g 1 (K, L, m/n) = | C (m) C (n)£ £ S 1 (K, L, i\ j') - cos 

N i'=l j'=i 

(N = 8,m = l-8,n = l~8) 

In this equation, a great m or n value represents a high hori- 
zontal or vertical frequency component in a macroblock. If 
the data representing those high-frequency components are re- 
moved, data with a low resolution can be generated easily and 
the camera image can be transmitted in a reduced amount. In 
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some type of synthesized image, a horizontal resolution re- 
quired may be different from a vertical resolution required. 
Even so, if the m and n values as thresholds for removing the 
excessive data are set independently, it is possible to sepa- 
rately control the resolutions both horizontally and verti- 
cally. 

FIGS. 5A and 5B schematically illustrate a correspon- 
dence between a synthesized image and a camera No. 1 image. 
Specifically, FIG. 5A illustrates a map for the camera No. 1 
image on the synthesized image, while FIG. 5B illustrates a 
map for the synthesized image on the camera No. 1 image. As 
shown in FIGS. 5A and 5B, the open circles A and B on the syn- 
thesized image are associated with the circles A' and B' on 
the camera No. 1 image. As used herein, the "map" means a 
lattice, which is drawn for convenience sake to clarify the 
correspondence between the synthesized and camera images. The 
lattice is drawn at regular pixel intervals in dashed lines 
for the synthesized image shown in FIG. 5A and in solid lines 
for the camera No. 1 image shown in FIG. 5B. The solid-line 
lattice on the camera No. 1 image also represents "blocks" as 
units for subsequent DCT compression. In the synthesized im- 
age shown in FIG. 5A, part of the camera No. 1 image map with 
a relatively high lattice density (e.g., the circle A) does 
not require so high a resolution for the camera No. 1 image 
data. On the other hand, part of the camera No. 1 image map 
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with a relatively low lattice density (e.g., the circle B) 
does require a high resolution for the camera No. 1 image 
data. That is to say, in the camera No. 1 image shown in FIG. 
5B, part of the synthesized image map with a relatively high 
5 lattice density (e.g., the circle B' associated with the cir- 
cle B) does require a high resolution for the camera No. 1 im- 
age data. However, part of the synthesized image map with a 
relatively low lattice density (e.g., the circle A' associated 

_ with the circle A) does not require so high a resolution for 
O 

10 the camera No. 1 image data. Also, the image data, not in- 

rj! eluded in the lattice representing the synthesized image map, 

5;;!: 

jtl is not necessary for the image synthesis. 

That is to say, the camera No. 1 image data may be clas- 

SI sified into three types of blocks BLl , BL2 and BL3 as shown in 

yj 15 FIGS. 6A and 6B. Specifically, the block BLl is not necessary 
□ 

for the image synthesis, the block BL2 requires high- 
resolution image data and the block BL3 needs only low- 
resolution image data. 

By reference to the data stored on the mapping table 220, 

20 the resolution specifier 260 derives resolutions data Rxv (K, 
L) and Rxh (K, L) for each macroblock, where x is the camera 
number (i.e., l^x^X) and h and v are horizontal and vertical 
directions, respectively. Hereinafter, it will be described 
in detail with reference to FIG. 7 how to derive the resolu- 

25 tion data Rxv (K, L) and Rxh (K, L) . 
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First, by reference to the data stored on the mapping ta- 
ble 220, it is determined whether or not any point in a mac- 
roblock of each camera image has its associated coordinates on 
the synthesized image. The block BL1 , including no points as- 
5 sociated with any sets of coordinates on the synthesized im- 
age, is not necessary for the image synthesis. Accordingly, 
the resolution data Rxv (K, L) and Rxh (K, L) for every point 
in the block BL1 should be zero. 

If any point (i', j') associated with coordinates (i, j) 
M3 10 on the synthesized image shown in FIG. 7 A exists on a mac- 
roblock of the camera image, then three more points, respec- 
rl tively associated with the coordinates (i+1, j), (i + 1, j + 1) 

J and (i, j + 1) on the synthesized image, are obtained for the 

D 

vj macroblock. Then, an area IM1, surrounded by the point (i', 

M 15 j') and the three other points, is defined as shown in FIG. 
Ms 7B. Next, using the vertical and horizontal sizes Lv and Lh 

(which are integers) of the area IM1 shown in FIG. 7C, the re- 
solution data Rxv (K, L) and Rxh (K, L) is derived by the fol- 
lowing equations : 
20 Rxv (K, L)=8/Lv 

Rxh (K, L)=8/Lh 
where the remainder should be rounded up. The resolution 
specifier 260 transmits the resolution data Rlv (K, L) through 
RXv (K, L) and Rlh (K, L) through RXh (K, L) obtained this way 
25 to the compressors 120 for the respective cameras 110 through 
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the transmission line 15. 

Strictly speaking, the area of the camera image occupied 
by the four pixels associated with the coordinates (i, j) on 
the synthesized image is not equal to the area IM but actually 
5 an area surrounded by the four points associated with the co- 
ordinates (i — 0.5, j — 0,5), (i — 0.5, j + 0.5), (i + 0.5, j — 0.5) 
and (i + 0.5, j+0.5), respectively, on the synthesized image. 
However, the latter area has almost the same shape as the area 
IM1 and the points associated with the coordinates (i, j), (i 
£ 10 +1/ j), (i+1, j + 1) and (i, j + 1) on the synthesized image 
p can be easily obtained by reference to the mapping table 220. 

fi For that reason, the resolution data Rxv (K, L) and Rxh (K, L) 

yl 

is herein derived based on the size of the area IM. 

In accordance with the resolution data Rxv (K, L) and Rxh 

u 

y 15 (K, L) transmitted, each compressor 120 removes unnecessary 
components from the signal gx (K, L, m, n) with respective 
frequencies for each macroblock of the associated camera im- 
age. Then, the compressor 120 transmits only the necessary 
components to the image processing section 20. Specifically, 

20 the compressor 120 removes components with m values greater 
than Rxv (K, L) and n values greater than Rxh (K, L) . In this 
case, the resolution data Rxv (K, L) and Rxh (K, L) of each 
and every point in the block BLl is zero. Accordingly, all 
the components in the block BLl are removed. 

25 FIG. 8 illustrates an exemplary internal configuration 
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for the compressor 120 associated with the camera No. 1. The 
compressor 120 includes three 8-line memories 121a, 121b and 
121c and can store image signals corresponding to 24 (=8X3) 
lines in total. 

5 When the resolution data Rlv (K, L) through RXv (K, L) 

and Rlh (K, L) through RXh (K, L) arrives by way of the trans- 
mission line 15 , each transmission adapter 130 selectively re- 
ceives the resolution data for the associated camera. In the 
illustrated example, the transmission adapter 130 receives the 

O 

^3 10 resolution data Rlv (K, L) and Rlh (K, L) associated with the 
?f camera No. 1. Then, the transmission adapter 130 stores the 

IS = 

resolution data on a resolution data memory 124. 

On the other hand, the camera 110 is sending out image 
signals as time-series data along the scan lines just like TV 
15 signals. Accordingly, a time t, at which a signal associated 
with a set of coordinates (i, j) on the screen are input, is 
given by 

t=Tpix * (i- (720+Bh)+j) 
where Tpix is a time per pixel and Bh is the number of hori- 
20 zontal blanking pixels. 

The image signals, sent out from the camera 110, are se- 
quentially stored on the respective 8-line memories 121a, 
121b and 121c. In this case, part of the image signal associ- 
ated with an i th line is stored at a mod (i, 24) th line. As 
25 used herein, mod (K, L) means the remainder obtained by divid- 
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ing K by L. 

When the first image signal has been stored to the 
eighth line on the first 8-line memory 121a, the next image 
signal for the ninth and succeeding lines starts to be stored 
on the second 8-line memory 121b. On the other hand, a digi- 
tal signal processor (DSP) 122 reads out the image signal, 
corresponding to the number 8 X 720 of pixels, as a signal SI 
(1, L, i', j') (where 1 ^ L ^ 90 , l^i'^8 and l^j'^8) for 
ninety 8X8 pixel macroblocks from the first 8-line memory 
121a. Then, the DSP 122 performs the DCT transform on each 
of these macroblocks, thereby calculating gl (1, L, m, n). 
Furthermore, by using the resolution data Rlv (1, L) and Rlh 
(1, L) stored on the resolution data memory 124, the DSP 122 
removes all components with m values greater than Rlv (1, L) 
and with n values greater than Rlh (1, L) from gl (1, L, m, 
n). 

Accordingly, at this point in time, the image signal SI 
(i, j) (where l^i^8 and l^j^ 720) stored on the first 8- 
line memory 121a is the following data sequence: 

gl (K = l, 1^L^90) 

(L= 1 ) : dl , d2 , dmn 
(L = 2) : dl, d2, dmn' 



<L = 90): dl, d2, dmn" 
where mn = Rlv (1, 1) - Rlh (1, 1), mn'=Rlv (1, 2) - Rlh (1, 
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2) and mn"=Rlv (1, 90) • Rlh (1, 90). As can be seen, the 
respective macroblocks include mutually different numbers of 
data items. That is to say, the image signal stored on the 
first 8-line memory 121a is transformed into a data sequence 
in which each macroblock includes a number (Rlv (1, L) • Rlh 
(1/ L)) of data items. This data sequence is once stored on a 
DCT data buffer memory 123 and then transmitted by the trans- 
mission adapter 130 to the image processing section 20 by way 
of the transmission line 15. 

On receiving this data sequence, the expander 240 per- 
forms inverse DCT (IDCT) transform on the data sequence, 
thereby restoring the image signal as SI' (1, L, i', j ' ) . 
Then, the expander 240 stores the restored image signal on 
the buffer memory 230. The restored image signal SI' (1, L, 
i', j') has already had its high-frequency components removed 
during the DCT transform and is equivalent to a signal ob- 
tained by subjecting the original signal SI (1, L, i', j') to 
low pass filtering (LPF) processing. 

Next, when the second image signal has been stored to 
the sixteenth line on the second 8-line memory 121b, the next 
image signal for the seventeenth and succeeding lines starts 
to be stored on the third 8-line memory 121c. On the other 
hand, the DSP 122 reads out the image signal, corresponding 
to the number 8 X 720 of pixels, as a signal SI (2, L, i', j') 
(where 1^L^90, l^i'^8 and l^j'^8) for ninety 8X8 pixel 
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macroblocks from the second 8-line memory 121b. Then, the 
DSP 122 performs the DCT transform on each of these mac- 
roblocks, thereby deriving gl (2, L, m, n) . Furthermore, by 
using the resolution data Rlv (2, L) and Rlh (2, L) stored on 
5 the resolution data memory 124, the DSP 122 removes all compo- 
nents with m values greater than Rlv (2, L) and with n values 
greater than Rlh (2, L) from gl (2, L, m, n) . 

Accordingly, at this point in time, the image signal 
stored on the second 8-line memory 121b is the following data 
y 10 sequence: 

0 gl (K = 2, 1^L^90) 

(L = l): dl, d2, dmn 

Ul (L = 2): dl, d2, dmn' 

n 



hi 15 (L=90): dl, d2 , dmn" 

£3 

Li where mn— Rlv (2, 1) - Rlh (2, 1), mn'—Rlv (2, 2) - Rlh (2, 

2) and mn"=Rlv (2, 90) - Rlh (2, 90). As can be seen, the 
respective macroblocks include mutually different numbers of 
data items. That is to say, the image signal stored on the 

20 second 8-line memory 121b is transformed into a data sequence 
in which each macroblock includes a number (Rlv (2, L) • Rlh 
(2, L)) of data items. In the same way, this data sequence is 
also transmitted through the transmission line 15 to the image 
processing section 20 and then restored by the expander 240 

25 into an image signal SI' (2, L, i', j')- 
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The same operation will be repeatedly performed until K 
=60. In this manner, image signals SI' (K, L, i', j') cor- 
responding to one frame are restored and then stored on the 
buffer memory 230. 
5 Also, the compressor 120 associated with each of the 

other cameras 110 also transmits compressed image data to the 
expander 240, which also restores image signals Sx' (K, L, 
i'f j') (where 2^x^X) and then stores them on the buffer 
memory 230. Then, the image synthesizer 210 produces a syn- 

10 thesized image from the respective camera image data stored 
on the buffer memory 230 by the known method. 

As described above, the camera section 10 of this em- 
bodiment compresses the image data of each camera image and 
then transmits just a part of the image data with specified 

15 resolutions in accordance with the correspondence between the 
synthesized and respective camera images as described on the 
mapping table 220. Accordingly, the transmission line 15 may 
have a smaller channel capacity. As a result, the monitoring 
system ensures more stabilized data transmission and can use 

20 an even less expensive transmission line. 

That is to say, the transmission lines for the respective 
cameras can be easily integrated into just one transmission 
line 15 as shown in FIG. 2. Thus, the transmission line 15 is 
much easier to mount on a vehicle and needs maintenance much 

25 less often. Alternatively, as shown in FIG. 9, the transmis- 
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sion line 15 may also be divided into two depending on the 
positions of the cameras mounted. That is to say, the moni- 
toring system of this embodiment can cope with various situa- 
tions flexibly enough. And this system is also much easier to 
install on a vehicle or requires much less maintenance. 

The transmission path does not have to be the transmis- 
sion line 15 but may also be a wireless path. This embodiment 
is also very effectively applicable to even a situation like 
that. That is to say, the channel capacity required can also 
be so small that the wireless transmission path is imple- 
mentable using less expensive parts. In addition, compared to 
the wired path, the wireless path can be installed on a vehi- 
cle and maintained even more easily. 

Also, part of a camera image that is not necessary for 
image synthesis (e.g., the block BL1 shown in FIG. 6B) needs 
to have resolution data Rxv (K, L) and Rxh (K, L) of (0, 0), 
and the number of data items in this block is zero. Accord- 
ingly, the amount of image data to be transmitted through the 
transmission line 15 can be reduced considerably. 

Moreover, in the image processing section 20, only a part 
of the image data with required resolutions should be stored 
for each camera image on the buffer memory 230. Thus, the 
buffer memory 230 may have a storage capacity much smaller 
than that of a known one. 

Furthermore, as soon as each camera image signal has been 
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stored on one of the 8-line memories 121a, 121b and 121c, the 
compressor 120 starts to perform the DCT transform and trans- 
mission. Accordingly, the compression and transmission causes 
a minimum signal propagation delay. 
5 In addition, the operation of compressing each camera im- 

age may also have its modes changed depending on the type of 
the synthesized image to be produced by the image processing 
section 20. In that case, every time the mapping tables for 
use in image synthesis are changed, the resolution specifier 

MS 10 260 should newly obtain the resolution data Rxv (K, L) and Rxh 

OB 

(K, L) and then transmit it to the compressors 120 for the re- 

H spective cameras 110 through the transmission line 15. 

yy 
m 

Alternatively, resolution data sets corresponding to the 

i J 

Zj respective mapping tables may also be stored beforehand on a 

yj 15 ROM, for example, provided for the resolution specifier 260. 
y= In that case, the selection signal should also be input to the 

resolution specifier 260 so that the resolution specifier 260 
can change the resolution data sets when the mapping tables 
are changed in response to the selection signal. Then, there 
20 is no need to perform the process step of newly obtaining the 
resolution data every time the mapping tables are changed. 

As another alternative, the compressor 120 may also in- 
clude a memory for pre-storing the resolution data sets, cor- 
responding to the respective mapping tables, thereon. Then, 
25 every time the mapping tables are changed, the image process- 
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ing section 20 has only to send the ID of the mapping table 
newly selected to the compressors 120. Alternatively, instead 
of getting the mapping table ID sent from the image processing 
section 20, the selection signal may also be input to the re- 
5 spective compressors 120 in the camera section 10 so that the 
compressors 120 can change the resolution data sets every time 
the mapping tables are changed. 

In the foregoing embodiment, a DCT transform is supposed 
to be carried out to compress the image data. Optionally, the 
10 DCT transformed data may be further subjected to quantization 
in compliance with the JPEG, for example. Also, similar ef- 
fects are attainable by a wavelet or Fourier transform, not 
just by the DCT transform. 



hi 15 Elimination of aliasing distortion 



In addition, according to this embodiment, it is possible 
to easily avoid an undesirable situation where the synthesized 
image has its quality lowered by aliasing distortion. 

In the example shown in FIGS. 7A and 7B, the area IM1 on 

20 the camera image, corresponding to the coordinates (i, j) on 
the synthesized image, has a size greater than that of one 
pixel of the camera image. That is to say, the area IM1 con- 
tains a number of pixels. To obtain an optically accurate 
synthesized image, a weighted average of the data represented 

25 by those pixels contained in the area IMl should be obtained 
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as the pixel data corresponding to the coordinates (i, j). 
Specifically, the pixel data S (i, j) of the synthesized image 
should be given by 

S (i, j) = Z 7 P • (SI <i p , j p ) ) 
where SI (i p , j p ) is the camera pixel data, p is the number 
of camera pixel and y p is a coefficient that has been de- 
termined considering the percentage of the pixel included 
in the area IM1 . In that case, however, to obtain a sig- 
nal corresponding to one pixel of the synthesized image, 
the computation should be carried out with signals corre- 
sponding to multiple pixels read out from the buffer memory 
230. This method is far from being practical because the 
image synthesizer 210 must perform a huge amount of compu- 
tation . 

For that reason, a so-called "nearest" approximation 
method has been used in the art. According to the nearest 
method, only a signal corresponding to just one pixel located 
nearest to the center of the area IM1 is used as the signal 
corresponding to the coordinates (i, j). 

The nearest method, however, often results in aliasing 
distortion, which is typically observed when a signal that was 
sampled at a high rate is sub-sampled at a low sampling rate 
without cutting off the high-frequency components thereof. 
That is to say, if simplified image synthesis is carried out 
by the nearest method using a high-resolution image signal as 
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it is, then the aliasing distortion might arise in part of the 
image that may have a low resolution, thus possibly lowering 
the quality of the synthesized image. 

To eliminate the aliasing distortion, those unwanted 
high-frequency components could be cut off in advance by get- 
ting the image signal of each camera pre-processed by its as- 
sociated low-pass filter (LPF) 51 as shown in FIG, 10. 

In that case, however, each LPF 51 should have its fre- 
quency characteristics controlled adaptively to every part of 
the associated camera image, because a resolution necessary 
for a part of the camera image might be different from one re- 
quired for another part thereof. That is to say, the LPF 51 
should have a rather complicated circuit configuration. On 
the other hand, if an LPF exhibiting constant frequency char- 
acteristics is used, then the LPF may have a simple circuit 
configuration. But the LPF automatically cuts off the high- 
frequency components from even a part requiring a high resolu- 
tion. Accordingly, the resultant synthesized image cannot have 
the required resolutions fully. 

In contrast, according to this embodiment, the aliasing 
distortion much less likely arises in the low-resolution image 
portion, although the aliasing distortion often occurs there 
normally. This is because the compressor 120 has already re- 
moved those high-frequency components from that image portion 
according to the size of the area IM1. That is to say, the 
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camera image signal, restored by the expander 240, is equiva- 
lent to a version of the original signal that has been sub- 
jected to the LPF processing adaptively depending on the 
resolutions required. Thus, the monitoring system of this em- 
5 bodiment can suppress the aliasing distortion sufficiently and 
can improve the quality of the resultant synthesized image 
significantly without using those overly complicated LPFs . 

In the foregoing example, the present invention is ap- 
plied to the nearest method. Alternatively, the present in- 

O 

p 10 vention is equally applicable to any other method. For 
example, this invention is obviously applicable to a bilinear 
method in which linear interpolation is carried out using sig- 
nals associated with four pixels that surround the center of 
the area IMl corresponding to the coordinates (i, j). 
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Alternative compression method 
In the foregoing embodiment, the resolution specifier 
260 obtains the resolution data Rxv (K, L) and Rxh (K, L) us- 
ing the horizontal and vertical sizes Lh and Lv of the area 
20 IMl shown in FIG. 7C. Alternatively, the resolution data may 
also be obtained by a different method. 

Like FIG. 7B, FIG. 11A also illustrates the area IMl oc- 
cupied by one pixel of the synthesized image on the camera im- 
age. In this alternative method, the area IMl is placed at 
25 the center of an 8X8 macroblock as shown in FIG. 11B and a 
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coefficient y p is obtained for each pixel while consider- 
ing the percentage of the pixel included in the area IM1. 
When a weighted average of multiple pixels included in a cam- 
era image is used to produce an optically accurate synthesized 

image, this coefficient y p is used to weight each of those 
pixels . 

Specifically, a coefficient r (i', j') is expanded into 
a coefficient for the 8X8 macroblock while supposing the 
center of the area IM1 to be represented by (i', j ' ) = ( 1 , 1)- 
Then, as in an image signal, the coefficient is subjected to 
a DCT transform, thereby obtaining a transform coefficient h 
(K, L, m, n). This transform coefficient h (K, L, m, n) is a 
DCT transform coefficient representing the LPF characteris- 
tics shown by the coefficient y p and has smaller high- 
frequency components . 

Thus, the upper limits of the m and n values in the 
transform coefficient h (K, L, m, n) with a predetermined 
threshold value or more are defined as the values of the reso- 
lution data Rxv (K, L) and Rxh (K, L). The resolution speci- 
fier 260 transmits not only the resolution data Rxv (K, L) and 
Rxh (K, L) but also the transform coefficient h (K, L, m, n) 
with the predetermined threshold value or more to the com- 
pressor 120. 

In response, the compressor 120 removes the high- 
frequency components from the DCT transformed data gx (K, L, 
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m, n) of the image signal using the resolution data Rxv (K, 
L) and Rxh (K, L). Next, the compressor 120 multiplies the 
remaining DCT transformed data gx (K, L, m, n) by a transform 
coefficient, hx (K, L, m, n) and then sends the product to the 
5 expander 240 . 

In accordance with this method, a signal restored by the 
expander 240 through IDCT transform is approximately equal to 
a version of the original signal that has been subjected to 
LPF processing using the coefficient 7 p . Thus, compared to 

O 

yj io the first embodiment, a more optically accurate synthe- 
Jj^ sized image can be produced. 

s . a 

It should be noted that the resolution data Rxv (K, L) 

* = and Rxh (K, L) is also definable without using the area IMl. 

□ 

%y For example, a block unnecessary for image synthesis may have 

yj 15 resolution data Rxv (K, L) and Rxh (K, L) of zero, while a 

□ 

block necessary for image synthesis may have resolution data 
Rxv (K, L) and Rxh (K, L) of a fixed non-zero value. 

EMBODIMENT 2 

20 FIG. 12 is a block diagram illustrating a configuration 

for a monitoring system according to a second embodiment of 
the present invention. The monitoring system shown in FIG. 
12 is basically the same as the counterpart shown in FIG. 1. 
In FIG. 12, each component also shown in FIG. 1 is identified 

25 by the same reference numeral and the detailed description 
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thereof will be omitted herein. 

The system shown in FIG. 12 is different from the coun- 
terpart shown in FIG. 1 in that the image processing section 
20A of the system shown in FIG. 12 further includes readout 
5 controller 270 and sync signal generator 280. In response to 
the output of the resolution specifier 260 , the readout con- 
troller 270 generates a readout control signal for each camera 
110A. The sync signal generator 280 generates a sync signal 
for each camera 110A. The readout control and sync signals, 
CI io output from the readout controller 270 and sync signal genera- 
jjf tor 280 , respectively, are transmitted from the transmission 

fi adapter 250 to the camera sections 10A by way of the transmis- 

LP 

~" sion line 15. The readout control and sync signals transmit - 

C3 

rji ted will be sent to the compressors 120A and cameras 11 OA by 

hj 15 way of the respective transmission adapters 130. 

O 

Synchronously with the rise of the sync signal transmit- 
ted, each camera 110A captures a frame picture. In the illus- 
trated embodiment, each camera 110A is supposed to capture 
image data consisting of 720X480 pixels every 1/60 second. 

20 FIG. 13 is a block diagram illustrating an exemplary in- 

ternal configuration for the camera 110A and compressor 120A. 
As shown in FIG. 13, the camera 110A includes: imager (e.g., 
CCD in this embodiment) 111; optical system 112 with a lens; 
readout/synchronization controller 113; and ADC/processor 114. 

25 The ADC/processor 114 performs an analog-to-digital conversion 
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on the output of the imager 111 and then various types of sig- 
nal processing (including color separation) on the resultant 
digital signal. 

The imager 111 includes imaging plane and storage each 
corresponding to one frame picture. First, incoming light is 
incident through the optical system 112 onto the imaging plane 
of the imager 111 and sensed as a light intensity signal 
thereon. Then, a phototransistor associated with each pixel 
converts the light intensity signal into an electrical charge 
signal. The electrical charge signal will be stored for one 
frame interval (e.g., 1/60 second in this embodiment) and then 
transferred as an image signal to the storage. 

In a normal CCD, the image signal, which has been accu- 
mulated on the storage, is sequentially read out along the 
scan lines. Typically, the upper left corner of one frame 
picture is read out first. In the meantime, another light in- 
tensity signal, which has just been sensed by the imaging 
plane, is converted into another electrical charge signal, 
which is then stored as an image signal for the next frame. 
In contrast, according to this embodiment, the imager 111 can 
read the image signal either from the upper left or lower left 
corner of one frame picture. And the order of reading the im- 
age signal is controlled by the readout/synchronization con- 
troller 113. 

In the second embodiment, the image processing section 
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20A controls the order, in which the image data obtained by 
each of the cameras 110A is read out, in accordance with the 
correspondence between the synthesized and respective camera 
images . 

5 FIG. 14 illustrates an exemplary order in which the im- 

age signal is read out from each of the cameras 110A. In 
FIG- 14, the areas CA1 ' through CA6 ' of the respective camera 
images are the same as the counterparts shown in FIG. 4B and 
are all necessary for image synthesis. As shown in FIG. 14, 
r^j 10 the lower part of each camera image is an area necessary for 
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the image synthesis, while the upper part thereof is unneces- 
sary for the image synthesis. 

In the illustrated embodiment, the readout controller 
270 generates a readout control signal instructing that the 

15 image signal captured by the camera No. 1, 3 or 5 should be 
read out from the upper left corner of one frame picture and 
that the image signal captured by the camera No. 2, 4 or 6 
should be read out from the lower left corner thereof as indi- 
cated by the arrows in FIG. 14. In response to the readout 

20 control signal 125 transmitted, the readout /synchronization 
controller 113 for each camera 110A controls the imager 111 as 
shown in FIG. 13. The output of the imager 111 is A-D con- 
verted and then subjected to various types of signal process- 
ing like color separation by the ADC/processor 114. Then, the 

25 digital signal processed is sequentially sent as image signals 
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to the 8-line memories 121 in the compressor 120A. 

As for the image signal captured by the camera No. 1, 3 
or 5, a signal corresponding to coordinates (i, j) on a syn- 
thesized image is sent at the time t given by 
5 t = Tpix-(i-(720+Bh)+j) 

as in the first embodiment. On the other hand, the image 
signal captured by the camera No. 2, 4 or 6 is scanned 
from the lower left corner of one frame picture. Accord- 
ingly, unlike the first embodiment, a signal corresponding 
10 to coordinates (i, j) on a synthesized image is sent at the 
time t given by 

t=Tpix ■ ((480-i) • (720+Bh)+j) 
where Tpix is a time per pixel and Bh is the number of 
horizontal blanking pixels. 
15 Accordingly, as in the first embodiment, the first 

image signal captured by the camera No. 1 is converted 
into the following data sequence in which the respective 
macroblocks have mutually different numbers of data items: 
gl (K = l, 1^L^90) 
20 (L = l): dl, d2, dmn 

(L = 2) : dl, d2, — , dmn' 



(L = 90) : dl, d2, 
where mn = Rlv (1, 1) • Rlh 
25 2 ) and mn"=Rlv (l, 90) 



dmn" 

(1, 1), mn'=Rlv (1, 2) -Rlh (1, 
• Rlh (1, 90). in contrast, the 
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first image signal captur 
verted into the following 
spective macroblocks have 
data items : 

g2 (K = 60, 1^L^90) 

(L=l) : dl, d2, — , 
(L = 2): dl, d2, — , 



d by the camera No, 2 is con- 
data sequence in which the re- 
mutually different numbers of 

dmn 
dmn ' 



(L = 90): dl f d2, dmn" 
where mn=R2v (60, 1) • R2h (60, 1), mn ' = R2v (60, 2) • 
R2h(60, 2) and mn" — R2v (60, 90) • R2h ( 60, 90 ). 

Next, it will be described how much data should be trans- 
mitted through the transmission line 15 when compressed image 
signals are sent out from the respective cameras 110A. 

As for the first image signal sent out from the camera 
No. 1, 3 or 5 for the first eight lines of one frame picture, 
mn through mn" are almost zero, i.e., only data with almost 
zero quantity is transmitted therefrom. On the other hand, 
the first image signal transmitted from the camera No. 2, 4 
or 6 for the first eight lines of one frame picture has a 
considerable amount of data necessary for image synthesis. 
As for the last image signal sent out from the camera No. 2, 
4 or 6 for the last eight lines of one frame picture, mn 
through mn" are almost zero, i.e., only data with almost zero 
quantity is transmitted therefrom. On the other hand, the 
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last image signal transmitted from the camera No, 1, 3 or 5 
for the last eight lines of one frame picture has a consider- 
able amount of data necessary for image synthesis. 

As can be seen, if the image signal is read out from the 
5 camera No. 1, 3 or 5 in the order opposite to that of the 
camera No. 2, 4 or 6, then the amount of data transmitted 
through the transmission line 15 can be dispersed with time. 
Accordingly, the transmission line 15 may have a relatively 
small channel capacity. 
10 Even for the first embodiment, the amount of data trans- 

it mitted through the transmission line 15 can also be dispersed 

jTjj with time. For example, the DCT data buffer memory 123 in 

J the compressor 120 may have so large a capacity that the data 

O 

Sj transmission is controllable on the transmitting end of the 

I t 

yj 15 DCT data buffer memory 123. Then, the amount of data trans- 
mitted can be dispersed with time. 

In that case, however, a huge amount of data should be 
stored on the DCT data buffer memory 123 albeit temporarily. 
Thus, the data transmitted from the camera 110 to the image 

20 processing section 20 is delayed for a longer time. Gener- 
ally speaking, the driver, or the user of this monitoring 
system, drives his or her vehicle while looking at the syn- 
thesized image on the screen. Accordingly, considering the 
response speed at outputting the synthesized image, such a 

25 delay should preferably be as short as possible. Thus, the 
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second embodiment is more advantageous than the first embodi- 
ment in this respect. 

As described above, according to the second embodiment, 
an image signal captured by each camera is read out in the 
> order that has been controlled in accordance with the corre- 
spondence between the synthesized and camera images. Thus, 
the amount of data transmitted can be dispersed with time 
without decreasing the response speed at outputting the syn- 
thesized image. 

10 It should be noted that the image signal readout orders 

shown in FIG. 14 are just illustrative ones. Accordingly, 
the readout control may also be performed so that the image 

signal is read out from the upper left corner of a picture 

□ 

SI taken by the camera No. 1, 2 or 3 and from the lower left 

□ 

W 15 corner of a picture taken by the camera No . 4 , 5 or 6 , for 
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example. Furthermore, if the correspondence between the syn- 
thesized and camera images has changed, then the image signal 
readout order may also be changed. In the foregoing embodi- 
ment, the imager 111 is so constructed as to provide two types 
20 of readout orders. Alternatively, the imager 111 may also re- 
alize three or more types of readout orders. Then, an appro- 
priate one of the readout orders should be selected depending 
on the correspondence between the synthesized and camera im- 
ages . 

25 In the foregoing embodiment, a CCD is supposed to be 



40 



m 



m 



p 



used as the imager. Alternatively, a CMOS device may also be 
used as the imager. A camera including a CCD outputs an im- 
age signal corresponding to the entire frame along the scan 
lines. On the other hand, a camera including a CMOS device 
5 can selectively output an image signal corresponding to just 
a part of one frame (e.g., a rectangular area). Accordingly, 
if a camera including a CMOS device is used, the image signal 
captured by each camera can be read out in a more finely con- 
trolled order. As a result, the amount of data transmitted 
10 can be dispersed with time more efficiently. 

EMBODIMENT 3 

FIG. 15 is a block diagram illustrating a configuration 
for a monitoring system according to a third embodiment of 

15 the present invention. The monitoring system shown in FIG. 
15 has basically the same configuration as the counterpart of 
the first embodiment shown in FIG. 1. 

The third embodiment of the present invention is differ- 
ent from the first embodiment in that the system of the third 

20 embodiment cuts down the amount of each camera image by re- 
moving the image data of an area that should be unnecessary 
for image synthesis, not by compressing the image data. For 
that purpose, the image processing section 20B of the third 
embodiment includes an area specifier 290 as alternative cut- 

25 down mode selecting means instead of the resolution specifier 
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260. The area specifier 290 specifies the area that should be 
necessary for image synthesis for each camera image. Also, 
the camera section 10B of the third embodiment does not in- 
clude the compressors 120 but selectors 140. Each of the se- 
5 lectors 140 removes the image data from the entire area of the 
associated camera image but the area specified by the area 
specifier 290. Furthermore, since this system performs no im- 
age compression, the image processing section 20B does not in- 
clude the expander 240. 
10 In accordance with the correspondence between the syn- 

thesized and camera images as described on the mapping table 
220, the area specifier 290 specifies the area that should be 
necessary for image synthesis for each camera image. The in- 
formation representing the areas specified is transmitted from 
yj 15 the transmission adapter 250 to the respective selectors 140 
in the camera section 10B by way of the transmission line 15. 

In response to the area information provided, each se- 
lector 140 selectively outputs the data about only the neces- 
sary area of the associated camera image. The image data 
20 output is transmitted from the associated transmission 
adapter 130 to the image processing section 20B by way of the 
transmission line 15. In the image processing section 20B, 
the image data transmitted is stored on the buffer memory 230 
first. Then, by reference to the data stored on the mapping 
25 table 220, the image synthesizer 210 produces a synthesized 
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image from the image data stored on the buffer memory 230. 

FIGS. 16A and 16B schematically illustrate a relation- 
ship between the image taken by the camera No. 1 and an area 
ANE necessary for image synthesis. As shown in FIG. 16B, the 
5 area ANE necessary for image synthesis can be obtained from 
the location of the synthesized image map on the camera No. 1 
image shown in FIG. 16A. For example , the area specifier 290 
obtains a rectangular area ANl including the necessary area 
^ ANE as shown in FIG. 17A, and then outputs the coordinates 

^ 10 (Is, Js) and (Ie, Je) at the upper left and lower right cor- 
m ners of the rectangular area ANl as information representing 

i 3 i the area ANl. In the same way, the area specifier 290 also 
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defines a similar rectangular area, which should be necessary 
in producing a synthesized image, for each of the other camera 

15 images and then outputs the coordinates at the upper left and 
lower right corners of the rectangular area obtained. 

FIG. 18 is a block diagram illustrating an exemplary in- 
ternal configuration for the selector 140. Hereinafter, it 
will be described with reference to FIG. 18 how the selector 

20 140 operates . 

The image signals output from the camera 110 are sequen- 
tially written onto three line memories 141a, 141b and 141c. 
On the other hand, area data, representing the areas specified 
for the respective camera images, has been transmitted through 

25 the transmission line 15 to the transmission adapters 130. 
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Each of the adapters 130 selectively takes only a part of the 
area data for the associated camera 110 and then gets the area 
data stored on an area data memory 144. 

When a predetermined amount of image data has been writ- 
5 ten on one of the line memories 141a, 141b or 141c , a DSP 142 
reads out the image data from the line memory 141a, 141b or 
141c. Next, the DSP 142 selects only a part of the image data 
that should be included in a rectangular area as defined by 
D the area data stored on the area data memory 144, and then 

EB 10 gets the selected image data stored on a data buffer memory 
HJ 143. The image data stored on the data buffer memory 143 is 

additionally provided with camera number, line numbers and 
headers representing the numbers of data items as follows : 
(Camera No. l)(Line No. l)(No. of Data Ml): dl , dMl 
15 (Camera No. l)(Line No. 2) (No. of Data M2 ) : dl, dM2 
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(Camera No. l)(Line No. 480) (No. of Data M480): dl, dM480 
Then, the image data with these identifiers is transmitted 
from the transmission adapter 130 to the image processing sec- 
20 tion 20B by way of the transmission line 15. 

The number Mi of data items for a line No. i is given by 
Mi — Je — Js + 1 if Is^i^le 
Mi — 0 otherwise 

That is to say, the amount of image data to be transmitted 
25 through the transmission line can be cut down considerably 
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compared to a known method that supposes that the image data 
should be all transmitted. 

In the foregoing embodiment, the area specifier 290 defi- 
nes a rectangular area ANl including an area ANE necessary for 
5 image synthesis and then transmits coordinate information, 
representing the rectangular area ANl, to the respective se- 
lectors 140. However, the present invention is not limited to 
such a specific embodiment. For example, as shown in FIG. 
^ 17B, each camera image may be digitized with the area ANE ne- 

10 cessary for image synthesis represented as "1" and the unne- 
cessary area AUN represented as "0". In that case, run length 

4ft? 
j r 

I ~ £ 

data, obtained by scanning the camera image along horizontal 
scan lines, may be transmitted. Then, in accordance with the 

t s 

SI run length data transmitted, each selector 140 selectively 

i ^ 

y 15 transmits only a part of its associated camera image data, 
r~ corresponding to the area ANE, to the image processing section 

20B. 

Also, as in the first embodiment, the operation of se- 
lecting a necessary area from each camera image may also have 

20 its modes changed depending on the type of a synthesized image 
to be produced by the image processing section 20B. In that 
case, every time the mapping tables for use in image synthesis 
are changed, the area specifier 290 should newly obtain the 
area data representing the areas necessary for the image syn- 

25 thesis and then transmit it to the selectors 140 for the re- 
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spective cameras 110 through the transmission line 15. 

Alternatively, area data sets, corresponding to the re- 
spective mapping tables, may also be stored beforehand on a 
ROM, for example, provided for the area specifier 290. In 
5 that case, the selection signal should also be input to the 
area specifier 290 so that the area specifier 290 can change 
the area data sets when the mapping tables are changed in re- 
sponse to the selection signal. Then, there is no need to 
perform the process step of newly obtaining the area data eve- 

10 ry time the mapping tables are changed. 

As another alternative, the selector 140 may also include 
a memory for pre-storing the area data sets, corresponding to 
the respective mapping tables, thereon. Then, every time the 
mapping tables are changed, the image processing section 20B 

15 has only to send the ID of the mapping table newly selected to 
the selectors 140. Alternatively, instead of getting the map- 
ping table ID sent from the image processing section 20B, the 
selection signal may also be input to the respective selectors 
140 in the camera section 10B so that the selectors 140 can 

20 change the area data sets every time the mapping tables are 
changed. 

It should be noted that the third embodiment of the pre- 
sent invention is combinable with the second embodiment. That 
is to say, the readout controller 27 0 and sync signal genera- 
25 tor 280 of the second embodiment may be added to the image 



46 



processing section 20B shown in FIG. 15 to control the camera 
image readout order in a similar manner. 

In the foregoing embodiments, the resolution or area 
specifier 260 or 290 is included in the image processing sec- 
tion 20 , 20A or 20B. Alternatively, the camera section 10 or 
10A may include means equivalent to the resolution specifier 
260 and the camera section 10B may include means equivalent to 
the area specifier 290. 

In that case, every time the types of synthesized images 
to be produced are changed, the image processing section 20, 
20A or 20B should transmit the data stored on the mapping ta- 
ble 220 newly selected to the camera section 10, 10A or 10B by 
way of the transmission adapter 250 and transmission line 15. 
Also, the camera section 10, 10A or 10B may also include a 
memory for pre-storing the resolution or area data sets, cor- 
responding to the respective mapping tables, thereon. Then, 
every time the types of synthesized images to be produced are 
changed, the image processing section 20, 20A or 20B has only 
to send the ID of the mapping table newly selected to the cam- 
era section 10, 10A or 10B. Alternatively, instead of getting 
the mapping table ID sent from the image processing section 
20, 20A or 20B, the selection signal may also be input to the 
camera section 10, 10A or 10B so that the camera section 10, 
10A or 10B can change the resolution or area data sets every 
time the mapping tables are changed. 
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Like the resolution or area specifier 260 or 290, the re- 
adout controller 270 and sync signal generator 280 of the sec- 
ond embodiment may also be included in the camera section 10, 
10A or 10B. 

In the foregoing embodiments, the camera images taken by 
the cameras are all supposed to be used for image synthesis* 
However, only some of those camera images might be necessary 
for image synthesis. In that case, the mode of cutting down 
the amount of data transmitted, including specifying the reso- 
lutions or areas required, should be selected for only those 
camera images necessary for image synthesis. 

Also, in the foregoing embodiments, the compressors 120 
or selectors 140 are provided as image data cutdown means for 
all the cameras. However, the compressors 120 or selectors 
140 may be provided for just some of the cameras. That is to 
say, some cameras may have the compressors 120 or selectors 
140 and the other cameras may have no compressors 120 or se- 
lectors 140. In that case, the mode of cutting down the 
amount of data transmitted, including specifying the resolu- 
tions or areas required, should be selected for only those 
cameras that do include the compressors 120 or selectors 140 
and are necessary for image synthesis. 

In the foregoing description, the monitoring system of 
the present invention is supposed to be applied to vehicles. 
However, the present invention is equally applicable to any 
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other types of moving objects including airplanes, boats and 
ships. Furthermore, cameras for the inventive monitoring sys- 
tem may be placed on a still object to be monitored, e.g., 
shops, residences and showrooms. Moreover, the positions and 
5 number of cameras to be mounted are not limited to the illus- 
trated ones . 

Furthermore, part or all of the functions of the inven- 
tive monitoring system may be implementable by either dedicat- 
ed hardware or software programs. It is also possible to use 
10 a storage or transmission medium that stores thereon a program 
for getting the inventive image processing executed by a com- 
puter either partially or entirely. 

As described above, according to the present invention, 
p4 image data, captured by respective cameras, has its amount cut 

•siasr 

y 

j«! 15 down in a mode that has been selected in accordance with a 
correspondence between a synthesized image to be produced and 
the respective camera images. Then, the image data in the re- 
duced amount is transmitted from the cameras to an image proc- 
essing section by way of a transmission path. In this manner, 

20 the amount of image data to be transmitted can be cut down 
considerably without sacrificing the quality of the resultant 
synthesized image. As a result, the transmission path is im- 
plementable as a smaller number of less expensive transmission 
lines or even as a wireless path. That is to say, the trans- 

25 mission path is much easier to mount on a vehicle or requires 
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maintenance much less often. In addition, a buffer memory re- 
quired for the image processing section can also have its 
storage capacity reduced greatly. The amount of image data to 
be transmitted can be cut down either by compressing the image 
5 data according to the resolutions that should be necessary for 
image synthesis or by removing the image data entirely but the 
areas necessary for the image synthesis. 
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